BLOCKBENCH: A Framework for Analyzing Private Blockchains

Research Paper Presentation

Cesaire Tobias

2025-05-02

Presentation Overview

  1. Introduction to Private Blockchains
  2. The Need for BLOCKBENCH
  3. Framework Architecture
  4. Benchmark Workloads
  5. Evaluation Methodology
  6. Performance Results
  7. Key Findings
  8. Implications for Industry
  9. Future Directions
  10. Conclusion

Introduction to Private Blockchains

View Blockchain Types Diagram in Appendix

  • Private Blockchains: Permissioned networks where participants are identified
  • Target Applications: Enterprise systems, consortium networks, supply chains
  • Key Platforms:
    • Ethereum
    • Hyperledger Fabric
    • Parity

The Need for BLOCKBENCH

View Need for Standard Framework Diagram in Appendix

  • Lack of Standard Evaluation Framework
  • Difficult to Compare Different Systems
  • Gap Between Industry Claims and Reality
  • No Baseline Comparison with Traditional Databases

Framework Architecture

View BLOCKBENCH Framework Diagram in Appendix

  • Layered Approach:
    • Consensus Layer
    • Data Model Layer
    • Execution Layer
  • Benchmark Interface
  • Metrics Focus: Throughput, Latency, Scalability, Fault Tolerance

Benchmark Workloads

View Benchmark Workloads Diagram in Appendix

Workload Description Tests
YCSB Key-value operations Read/write/scan operations
Smallbank Banking transactions Account transfers, balance checks
EtherId Name registration Complex state operations
Donothing Empty blocks Consensus overhead

Evaluation Methodology

View Evaluation Methodology Diagram in Appendix

  • Systems Under Test:
    • Ethereum (PoW)
    • Hyperledger Fabric (PBFT)
    • Parity (PoA)
    • H-Store (traditional distributed database)
  • Controlled Environment: 48-node cluster
  • Statistical Rigor: Multiple runs, confidence intervals
  • Parameter Optimization: Each system configured optimally

Performance Results: Throughput

View Throughput Comparison Chart in Appendix

  • Blockchain vs Database:
    • H-Store: 10,000+ TPS
    • Private Blockchains: 1,000-3,000 TPS peak
  • Platform Comparison:
    • Ethereum: Moderate but consistent
    • Hyperledger: Higher peaks, more variability
    • Parity: Performance varies by workload

Performance Results: Latency

View Latency Comparison Chart in Appendix

  • Transaction Confirmation Time:
    • H-Store: Milliseconds
    • Ethereum: 3-10 seconds
    • Hyperledger: 100-800 milliseconds
    • Parity: 200-1000 milliseconds
  • Consensus Protocol Impact:
    • PoW (Ethereum): Highest latency
    • PBFT (Hyperledger): Lower but scaling issues
    • PoA (Parity): Varies with validator count

Performance Results: Scalability

View Scalability Analysis Chart in Appendix

  • Negative Scalability:
    • Performance degrades as nodes increase
    • 30-50% reduction when doubling nodes
  • Consensus Bottlenecks:
    • PBFT: O(n²) message complexity
    • PoW: Network propagation overhead
  • Contrast with Databases:
    • H-Store: Near-linear scaling (positive)

Layer-Specific Findings

View Layer Contribution Chart in Appendix

  • Consensus Layer: 80-90% of total latency
  • Data Model Layer:
    • Account-based (Ethereum): Better for read-heavy workloads
    • UTXO-based (Hyperledger): Better for write-intensive applications
  • Execution Layer:
    • 10-100x overhead compared to native execution
    • Inefficient memory management
    • Limited optimization capabilities

Key Findings and Implications

View Key Findings Diagram in Appendix

  1. Performance Gap: Private blockchains significantly underperform databases
  2. Negative Scalability: Adding nodes decreases performance
  3. Execution Inefficiency: Smart contracts introduce major overhead
  4. Consensus Dominance: Protocol choice is the primary performance determinant
  5. Data Model Impact: Selection affects workload suitability

Implications for Industry

View Industry Blockchain Readiness Chart in Appendix

Industry Requirements Blockchain Readiness
Financial Services High throughput, low latency Limited - Hybrid solutions recommended
Supply Chain Scalability, moderate throughput Moderate - Careful design needed
Healthcare Confidentiality, moderate latency Promising - With optimization
Real Estate Security, lower throughput Good fit - Current performance adequate

Conclusion

View Path Forward Diagram in Appendix

  • BLOCKBENCH Contribution: First systematic framework for evaluation
  • Current State: Promising technology with significant limitations
  • Path Forward:
    • Consensus protocol innovation
    • Execution optimization
    • Data model hybridization
    • Architecture rethinking

Thank You

Questions?

Appendix

The following slides contain supplementary information including charts, diagrams, definitions, references, and more detailed results from the paper.

Return to Presentation Overview

BLOCKBENCH Paper

View BLOCKBENCH Paper

Charts and Diagrams

Blockchain Types

%%{init: {'theme': 'default', 'themeVariables': { 'fontSize': '22px', 'primaryColor': '#7C0000', 'primaryBorderColor': '#7C0000', 'primaryTextColor': '#fff', 'secondaryColor': '#006100', 'secondaryBorderColor': '#006100', 'secondaryTextColor': '#fff' }}}%%
graph TD
    classDef mainNode fill:#f9f9f9,stroke:#333,stroke-width:2px;
    classDef publicNode fill:#d1f0ff,stroke:#0077b6,stroke-width:2px;
    classDef privateNode fill:#ffedd8,stroke:#e67700,stroke-width:2px;
    
    A[Blockchain Types] --> B[Public Blockchain]
    A --> C[Private Blockchain]
    B --> B1[Open Access: Anyone can join]
    B --> B2[Examples: Bitcoin, Ethereum]
    B --> B3[High Decentralization]
    C --> C1[Permissioned: Known participants]
    C --> C2[Examples: Hyperledger, Parity]
    C --> C3[Enterprise Use: Supply chain, finance]
    
    class A mainNode;
    class B,B1,B2,B3 publicNode;
    class C,C1,C2,C3 privateNode;

Figure A1: Types of blockchain systems and their characteristics

Return to Introduction to Private Blockchains

Need for Standard Framework

%%{init: {'theme: 'default', 'themeVariables': { 'fontSize': '22px', 'primaryColor': '#7C0000', 'primaryBorderColor': '#7C0000', 'primaryTextColor': '#fff', 'secondaryColor': '#006100', 'secondaryBorderColor': '#006100', 'secondaryTextColor': '#fff' }}}%%
graph TD
    classDef mainNode fill:#f9f9f9,stroke:#333,stroke-width:2px;
    classDef problemNode fill:#ffd6d6,stroke:#c81d25,stroke-width:2px;
    classDef solutionNode fill:#d1f5d3,stroke:#2b9348,stroke-width:2px;
    
    A[Blockchain Systems] --> B[Vendor Claims]
    A --> C[Custom Evaluations]
    B --> D[Inconsistent Metrics]
    C --> D
    D --> E[No Standard Framework]
    E --> F[Need for BLOCKBENCH]
    
    class A mainNode;
    class B,C,D,E problemNode;
    class F solutionNode;

Figure A2: Problem space that motivated the development of BLOCKBENCH

Return to The Need for BLOCKBENCH

BLOCKBENCH Framework

%%{init: {'theme': 'default', 'themeVariables': { 'fontSize': '22px', 'primaryColor': '#7C0000', 'primaryBorderColor': '#7C0000', 'primaryTextColor': '#fff', 'secondaryColor': '#006100', 'secondaryBorderColor': '#006100', 'secondaryTextColor': '#fff' }}}%%
graph TD
    classDef mainNode fill:#f5f5f5,stroke:#333,stroke-width:2px;
    classDef layerNode fill:#d1e7ff,stroke:#0077b6,stroke-width:2px;
    classDef sublayerNode fill:#e6f3ff,stroke:#4da6ff,stroke-width:2px;
    classDef interfaceNode fill:#ffdbb5,stroke:#fc7a08,stroke-width:2px;
    classDef metricsNode fill:#ffe8b5,stroke:#fcba08,stroke-width:2px;
    
    A[BLOCKBENCH Framework] --> B[Consensus Layer]
    A --> C[Data Model Layer]
    A --> D[Execution Layer]
    B --> B1[Transaction Agreement]
    C --> C1[State Management]
    D --> D1[Smart Contract Execution]
    A --> E[Benchmark Interface]
    E --> F[Metrics: Throughput, Latency, Scalability, Fault Tolerance]
    
    class A mainNode;
    class B,C,D layerNode;
    class B1,C1,D1 sublayerNode;
    class E interfaceNode;
    class F metricsNode;

Figure A3: Architecture of the BLOCKBENCH evaluation framework

Return to Framework Architecture

Benchmark Workloads Diagram

%%{init: {'theme': 'default', 'themeVariables': { 'fontSize': '22px', 'primaryColor': '#7C0000', 'primaryBorderColor': '#7C0000', 'primaryTextColor': '#fff', 'secondaryColor': '#006100', 'secondaryBorderColor': '#006100', 'secondaryTextColor': '#fff' }}}%%
graph TD
    classDef mainNode fill:#f5f5f5,stroke:#333,stroke-width:2px;
    classDef workloadNode fill:#daffb3,stroke:#459a30,stroke-width:2px;
    classDef detailNode fill:#f0ffdc,stroke:#6cbc49,stroke-width:2px;
    
    A[Workloads] --> B[YCSB]
    A --> C[Smallbank]
    A --> D[EtherId]
    A --> E[Donothing]
    B --> B1[Key-value operations]
    C --> C1[Banking transactions]
    D --> D1[Name registration]
    E --> E1[Empty blocks]
    
    class A mainNode;
    class B,C,D,E workloadNode;
    class B1,C1,D1,E1 detailNode;

Figure A4: Four benchmark workloads used in BLOCKBENCH evaluation

Return to Benchmark Workloads

Evaluation Methodology Diagram

%%{init: {'theme': 'default', 'themeVariables': { 'fontSize': '22px', 'primaryColor': '#7C0000', 'primaryBorderColor': '#7C0000', 'primaryTextColor': '#fff', 'secondaryColor': '#006100', 'secondaryBorderColor': '#006100', 'secondaryTextColor': '#fff' }}}%%
graph LR
    classDef stepNode fill:#e6f7ff,stroke:#0077b6,stroke-width:2px;
    
    A[Setup] --> B[Systems: Ethereum, Hyperledger, Parity, H-Store]
    B --> C[48-node Cluster]
    C --> D[Optimize Parameters]
    D --> E[Run Benchmarks]
    E --> F[Collect Metrics]
    F --> G[Statistical Analysis]
    
    class A,B,C,D,E,F,G stepNode;

Figure A5: Evaluation methodology workflow

Return to Evaluation Methodology

Throughput Comparison

Return to Performance Results: Throughput

Latency Comparison

Return to Performance Results: Latency

Scalability Analysis

Figure 1: ?(caption)

Return to Performance Results: Scalability

Layer Contribution

Return to Layer-Specific Findings

Key Findings

%%{init: {'theme': 'default', 'themeVariables': { 'fontSize': '22px', 'primaryColor': '#7C0000', 'primaryBorderColor': '#7C0000', 'primaryTextColor': '#fff', 'secondaryColor': '#006100', 'secondaryBorderColor': '#006100', 'secondaryTextColor': '#fff' }}}%%
graph TD
    A[Key Findings] --> B[Performance Gap]
    A --> C[Negative Scalability]
    A --> D[Execution Inefficiency]
    A --> E[Consensus Dominance]
    A --> F[Data Model Impact]

Figure A6: Key findings from the BLOCKBENCH study

Return to Key Findings and Implications

Industry Readiness

Return to Implications for Industry

Path Forward

%%{init: {'theme': 'default', 'themeVariables': { 'fontSize': '22px', 'primaryColor': '#7C0000', 'primaryBorderColor': '#7C0000', 'primaryTextColor': '#fff', 'secondaryColor': '#006100', 'secondaryBorderColor': '#006100', 'secondaryTextColor': '#fff' }}}%%
graph TD
    A[Path Forward] --> B[Consensus Innovation]
    A --> C[Execution Optimization]
    A --> D[Data Model Hybridization]
    A --> E[Architecture Rethinking]

Figure A7: Proposed directions for future blockchain development

Return to Conclusion

Detailed Results

YCSB Benchmark Results

  • Read Operations:
    • Ethereum: 300-400 ops/sec
    • Hyperledger: 1000-1200 ops/sec
    • Parity: 800-900 ops/sec
    • H-Store: 10,000+ ops/sec
  • Write Operations:
    • Generally 30-50% slower than reads across all platforms

Smallbank Benchmark Results

  • Transaction Throughput:
    • Ethereum: 250-300 TPS
    • Hyperledger: 700-850 TPS
    • Parity: 500-650 TPS
    • H-Store: 8,000-9,000 TPS
  • Consistency Testing:
    • All systems maintain ACID properties
    • Blockchain adds overhead for consistency guarantees

EtherId Benchmark Results

  • Complex Smart Contract Operations:
    • Ethereum: 40-50 ops/sec
    • Hyperledger: 100-150 ops/sec
    • Parity: 70-90 ops/sec
  • Execution Overhead Analysis:
    • VM execution: 40-60% of non-consensus time
    • State access: 15-25% of non-consensus time
    • Memory management: 10-20% of non-consensus time

Reference Materials

Blockchain Terminology

Term Definition
Blockchain A distributed digital ledger with a linked chain of blocks containing transaction records
Private Blockchain A permissioned blockchain where participants are known and authorized
Consensus Protocol Mechanism by which nodes agree on the state of the blockchain
Smart Contract Self-executing code that runs on the blockchain
Node A computer that participates in the blockchain network
Block A container of transactions added to the blockchain
Transaction An operation that changes the blockchain state

Consensus Protocols

Protocol Description Used By Characteristics
PoW (Proof of Work) Nodes solve cryptographic puzzles Ethereum High security, high latency, energy intensive
PBFT (Practical Byzantine Fault Tolerance) Multi-round voting process Hyperledger Fabric Lower latency, O(n²) message complexity
PoA (Proof of Authority) Authorized validators only Parity Low latency, centralization concerns

System Architecture Comparison

%%{init: {'theme': 'default', 'themeVariables': { 'fontSize': '22px', 'primaryColor': '#7C0000', 'primaryBorderColor': '#7C0000', 'primaryTextColor': '#fff', 'secondaryColor': '#006100', 'secondaryBorderColor': '#006100', 'secondaryTextColor': '#fff' }}}%%
graph TD
    A[Systems] --> B[Ethereum]
    A --> C[Hyperledger]
    A --> D[Parity]
    B --> B1[Account-based]
    B --> B2[PoW]
    B --> B3[EVM]
    C --> C1[UTXO-based]
    C --> C2[PBFT]
    C --> C3[Docker]
    D --> D1[Account-based]
    D --> D2[PoA]
    D --> D3[EVM]

Figure A8: Comparison of system architectures

System Data Model Consensus Execution Environment
Ethereum Account-based PoW EVM (stack-based VM)
Hyperledger UTXO-based PBFT Docker container
Parity Account-based PoA EVM (stack-based VM)

Future Research Directions

  • Layered Consensus Protocols: Separate consensus for different operation types
  • Just-In-Time Compilation: Smart contract optimization techniques
  • Sharding Approaches: Horizontal partitioning for scalability
  • Hybrid Architecture: Blockchain for trust, databases for performance
  • Hardware Acceleration: Specialized hardware for cryptographic operations

References

  1. Dinh, T. T. A., Wang, J., Chen, G., Liu, R., Ooi, B. C., & Tan, K. L. (2017). BLOCKBENCH: A framework for analyzing private blockchains. In Proceedings of the 2017 ACM International Conference on Management of Data (pp. 1085-1100).

  2. Nakamoto, S. (2008). Bitcoin: A peer-to-peer electronic cash system. Bitcoin.org

  3. Wood, G. (2014). Ethereum: A secure decentralised generalised transaction ledger. Ethereum project yellow paper.

  4. Androulaki, E., Barger, A., Bortnikov, V., et al. (2018). Hyperledger fabric: a distributed operating system for permissioned blockchains. In Proceedings of the Thirteenth EuroSys Conference (pp. 1-15).

Additional Definitions

Term Definition
Throughput Number of transactions processed per second
Latency Time from submission to confirmation of a transaction
Scalability System performance change as network size increases
EVM Ethereum Virtual Machine - execution environment for smart contracts
UTXO Unspent Transaction Output - data model used in Bitcoin and some other blockchains
Sharding Technique of partitioning data across multiple nodes
Gas Unit measuring computational work in Ethereum

Financial Software Engineering (ECO5040S) - Assignment 1